Image processing apparatus, image pickup apparatus, image processing method, and image processing program using compound kernel

ABSTRACT

An image processing apparatus that blurs a picked-up image based on distance information on a subject, comprises a restoration kernel acquisition section that acquires a restoration kernel as a kernel for eliminating degradation of an image; a blur kernel acquisition section that acquires a blur kernel as a kernel for blurring the image; a compound kernel acquisition section that acquires a compound kernel obtained by merging the restoration kernel with the blur kernel; and an image processing section that eliminates the degradation of the picked-up image and blurs the picked-up image by using the compound kernel.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to digital image processing, andparticularly relates to an image processing technology that performsblurring on an image.

2. Description of the Related Art

In recent years, there is proposed a compact digital camera capable ofobtaining an excellent blur by increasing the size of an image sensor.However, when the size of the image sensor is increased, the size of animage-forming optical system is also increased so that it becomesdifficult to reduce the size of the camera. To cope with this, JapanesePatent Application Laid-open No. 2000-207549 discloses a camera in whichblurring is performed by picking up an image that is nearly in deepfocus (a state where all areas are in focus), acquiring information onthe distance to a subject, and performing predetermined image processingon the picked-up image.

However, even when the deep-focus image is to be picked up, in reality,it is not possible to obtain an image which is perfectly focused in allareas due to influences of diffraction caused by an aperture and imageplane aberration, and a difference in object distance. Particularly in acompact digital camera, the influence of the diffraction or theaberration tends to be increased. To cope with this, Japanese PatentApplication Laid-open No. 2011-211663 discloses an invention in which animage degraded by the influence of the diffraction or aberration isrestored by digital processing.

In a case where blurring processing in which the focus position of apicked-up image is changed is performed, when the picked-up image is nota strictly deep-focus image, there arises a problem that the imagequality after the blurring is degraded, i.e., apart that should be infocus is blurred. In addition, there is a problem that, even in the partin focus, the image is degraded by the influence of the diffraction oraberration and a blurred image having high image quality cannot beobtained.

SUMMARY OF THE INVENTION

In order to solve the above problems, an object of the present inventionis to provide an image processing apparatus, an image pickup apparatus,an image processing method, and an image processing program capable ofgenerating a blurred image having high quality without the influence ofthe diffraction or aberration.

The present invention in its one aspect provides an image processingapparatus that blurs a picked-up image based on distance information ona subject, comprising:

a restoration kernel acquisition section that acquires a restorationkernel as a kernel for eliminating degradation of an image; a blurkernel acquisition section that acquires a blur kernel as a kernel forblurring the image; a compound kernel acquisition section that acquiresa compound kernel obtained by merging the restoration kernel with theblur kernel; and an image processing section that eliminates thedegradation of the picked-up image and blurs the picked-up image byusing the compound kernel.

The present invention in its another aspect provides an image processingapparatus that blurs a picked-up image based on distance information ona subject, comprising: a restoration kernel acquisition section thatacquires a restoration kernel as a kernel for eliminating degradation ofan image; a blur kernel acquisition section that acquires a blur kernelas a kernel for blurring the image; and an image processing section thateliminates the degradation of the picked-up image and blurs thepicked-up image by using the restoration kernel and the blur kernel,wherein the restoration kernel acquisition section acquires therestoration kernel associated with a distance and an angle of viewcorresponding to an area to be processed on the picked-up image, and theblur kernel acquisition section acquires the blur kernel associated withthe distance and the angle of view corresponding to the area to beprocessed on the picked-up image.

The present invention in its another aspect provides a non-transitoryrecording medium recording thereon an image processing program forcausing an image processing apparatus that blurs a picked-up image basedon distance information on a subject to execute the steps of: acquiringa restoration kernel as a kernel for eliminating degradation of animage; acquiring a blur kernel as a kernel for blurring the image;acquiring a compound kernel obtained by merging the restoration kernelwith the blur kernel; and eliminating the degradation of the picked-upimage and blurring the picked-up image by using the compound kernel.

The present invention in its another aspect provides a non-transitoryrecording medium recording thereon an image processing program forcausing an image processing apparatus that blurs a picked-up image basedon distance information on a subject to execute the steps of: acquiringa restoration kernel as a kernel for eliminating degradation of animage; acquiring a blur kernel as a kernel for blurring the image; andeliminating the degradation of the picked-up image and blurring thepicked-up image by using the restoration kernel and the blur kernel,wherein the restoration kernel associated with a distance and an angleof view corresponding to an area to be processed on the picked-up imageis acquired in the step of acquiring a restoration kernel, and the blurkernel associated with the distance and the angle of view correspondingto the area to be processed on the picked-up image is acquired in thestep of acquiring a blur kernel.

According to the present invention, it is possible to provide the imageprocessing apparatus, the image pickup apparatus, the image processingmethod, and the image processing program capable of generating theblurred image having high quality without the influence of thediffraction or aberration.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the configuration of an imageprocessing apparatus according to a first embodiment;

FIG. 2 is a flowchart showing the operation of the image processingapparatus according to the first embodiment;

FIG. 3 is a flowchart showing the operation of area generationprocessing in the first embodiment;

FIG. 4 is a view illustrating a map generated in the first embodiment;

FIG. 5 is a view illustrating composition of a calculated kernel in thefirst embodiment;

FIGS. 6A and 6B are views illustrating the area generation processing inthe first embodiment; and

FIG. 7 is a block diagram showing the configuration of an image pickupapparatus according to a second embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

(First Embodiment)

Hereinbelow, a description will be given of an image processingapparatus as one aspect of the present invention with reference to thedrawings. Like reference numeral in each drawing denote like means. Notethat the scope of the present invention is not intended to be limited toexamples shown in the description of the embodiments.

<System Configuration>

FIG. 1 is a system configuration view of an image processing apparatusaccording to a first embodiment.

An image processing apparatus 1 has a parameter input section 101, aparameter memory section 102, an image data input section 103, an imagememory section 104, a PSF memory section 105, and an area generationsection 106. In addition, the image processing apparatus 1 also has ablur kernel generation section 107, a restoration kernel generationsection 108, an image processing operation section 109, and a processedimage memory section 110.

The image processing apparatus according to the present embodiment maybe implemented by using dedicated circuits or may be implemented by acomputer. When the image processing apparatus is implemented by thecomputer, a program stored in an auxiliary storage device is loaded intoa main storage device and executed by a CPU, and the individual meansshown in FIG. 1 thereby function (the CPU, the auxiliary storage device,and the main storage device are not shown in the drawing).

The parameter input section 101 is an input section for inputtingparameters related to image pickup conditions such as a focus positionand an exposure, a threshold of an image degradation amount used in areageneration processing described later, and parameters required todetermine the size and shape of a blur to the image processingapparatus. The parameter memory section 102 is a memory for storing theparameters inputted from the parameter input section 101.

The image data input section 103 is an input section for inputting dataon an image picked up by image pickup means that is not shown, i.e.,data in which the picked-up image is represented by two-dimensionalbrightness values, and distance information corresponding to the imagedata to the image processing apparatus. A detailed description of thedistance information will be given later. The image memory section 104is a memory for storing the image data inputted to the image data inputsection 103.

The PSF memory section 105 is a memory for storing a PSF for imagerestoration, or data that can replace the PSF. The point spread function(PSF) mentioned herein is a function representing the spread of rayswhen an ideal point image passes through a target optical system. In thepresent embodiment, the PSF for restoring an image degraded byaberration or diffraction by means of deconvolution processing isstored. The distance to a subject and the angle of view have variousconditions, and hence the PSF memory section 105 stores a plurality ofthe PSFs corresponding to the individual conditions.

The area generation section 106 is means for generating a processingarea when image restoration and blurring processing are performed basedon the information stored in the parameter memory section 102, the imagememory section 104, and the PSF memory section 105. More specifically,the area generation section 106 determines an image area (an extendedarea in the present invention) of the picked-up image where theprocessing can be performed by using an identical blur kernel and anidentical restoration kernel. The area generation section 106 is animage area extension section in the present invention.

The blur kernel generation section 107 is means for generating the blurkernel corresponding to the image area determined by the area generationsection 106. The blur kernel mentioned herein is a kernel for performingfilter processing for blurring the image. The blur kernel generationsection 107 is capable of generating the blur kernel associated withconditions such as the object distance, the angle of view, an F-number,a focal length, and parameters related to the blur. The blur kernelgeneration section 107 is a blur kernel acquisition section in thepresent invention. The phrase “object distance” used in this descriptionindicates a distance to the subject.

The restoration kernel generation section 108 is means for determiningthe PSF corresponding to the image area generated by the area generationsection 106 based on the plurality of the PSFs stored in the PSF memorysection 105 and generating the restoration kernel as the kernel forperforming image restoration processing. The restoration kernelgeneration section 108 is a restoration kernel acquisition section inthe present invention.

The image processing operation section 109 is means for performingoperation processing that gives effects of the image restoration and theblurring to the image stored in the image memory section 104 based onthe information obtained from the area generation section 106, the blurkernel generation section 107, and the restoration kernel generationsection 108.

The processed image memory section 110 is a memory for storing theblurred image generated in the image processing operation section 109.

<Processing Flowchart>

Next, a detailed description will be given of the operation of the imageprocessing apparatus 1 with reference to a processing flowchart. FIG. 2is a flowchart showing the processing performed by the image processingapparatus 1.

In Step S201, the parameter input section 101 acquires the parametersrelated to the image pickup conditions, the threshold of the imagedegradation amount used in the area generation processing describedlater, and the parameters required for the blurring. The parameters maybe inputted by a user or may also be acquired from the image pickupmeans. Alternatively, pre-stored parameters may also be acquired. Theinputted parameters are stored in the parameter memory section 102. Byusing these parameters, the area generation processing, blur kernelgeneration processing, and restoration kernel generation processingdescribed later are executed.

In Step S202, the image data input section 103 acquires the image datapicked up by the image pickup means and the distance informationcorresponding to the image data or data convertible to the distanceinformation, and causes the image memory section 104 to store them. Thedata convertible to the distance information mentioned herein includesdirect distance information such as a relative distance from the eyepoint position of inputted image data and a position in globalcoordinates or the like, and parallax information generated by usingimage information from a plurality of the image pickup means.

In addition, the data convertible to the distance information may alsobe data for acquiring the distance by using the technology ofcomputational photography as an image-forming method using digitalprocessing.

For example, in a technology called depth from defocus (DFD), it ispossible to determine the object distance by acquiring a focus image anda defocus image and analyzing a blur amount. In addition, the dataconvertible to the distance information may be any data as long as thedata is for acquiring the object distance. For example, a defocus amountcalculated from the distance information and the focus position as oneof the parameters related to the image pickup conditions inputted inStep S201 may be used as the input data.

In Step S203, when the image restoration processing and the imageblurring processing are performed, processing for generating a new areaserving as a shift-invariant area is performed. The shift-invariant areamentioned herein denotes an area to which the identical kernel can beapplied in the image processing operation (the image restorationprocessing and the image blurring processing) described later.

In other words, the shift-invariant area is an area where thedegradation of the image quality falls within a permissible range evenwhen the processing is performed by using the identical kernel. In acase where the present step is not executed, it is necessary to performthe image processing operation for each area to be processed (e.g., foreach pixel held by the image). However, by executing the present step,it is possible to reduce the number of areas to be processed whileminimizing the degradation of the image quality.

Herein, a detailed description will be given of the content of the areageneration processing performed by the area generation section 106 inStep S203. FIG. 3 is a flowchart showing the processing in Step S203 indetail.

First, in Step S301, the generation of a map is performed. The map isthe data group of a plurality of the restoration kernels and the blurkernels assigned to different areas on the image.

FIG. 4 is a view two-dimensionally showing an example of the map. Therestoration kernel and the blur kernel to be used are assigned to eachof the areas to be processed preset on the picked-up image. Thereference numeral 401 denotes the kernel assigned to each area to beprocessed. The area to be processed may be an area including only onepixel and may also be an area having an arbitrary shape that includes aplurality of pixels. This map is the map before the area generationsection 106 performs the area generation processing.

The restoration kernel and the blur kernel assigned to each area to beprocessed differ depending on the actual image pickup conditions andoptical configuration, i.e., the object distance and the angle of view.The kernel generated by the operation every time the processing isperformed may be acquired or the kernel that is pre-calculated andstored may also be acquired.

It is possible to generate the blur kernel by using a plurality of thePSFs corresponding to the blurring conditions (the object distance, theangle of view, the intensity of the blur, and the like). The PSF used inthe generation of the blur kernel may be generated by the operation fromthe blurring conditions on an as needed basis, or the appropriate onemay also be selected from the PSFs that are pre-calculated and stored.With this arrangement, it is possible to impart the blur in line withthe actual optical system.

The blur kernel may not be necessarily generated by the PSF in line withthe actual optical system. For example, a shared blur kernel is storedand an objective blur kernel may be acquired by converting the sharedblur kernel by using the object distance and the angle of view as theparameters. Thus, the blur kernel to be used may be determined in anymanner in a desired form.

A kernel obtained by deconvoluting the blur kernel using the restorationkernel and merging the blur kernel with the restoration kernel may beassigned to each area to be processed (hereinafter this kernel isreferred to as a calculated kernel). FIG. 5 is a view showing thecalculated kernel obtained by deconvoluting the blur kernel using therestoration kernel in the form of a two-dimensional figure. In otherwords, the figure shown in FIG. 5 represents an area formed of a filtermatrix corresponding to the calculated kernel. Note that the calculatedkernel is a compound kernel in the present invention.

The calculated kernel assigned to the area to be processed also differsdepending on the distance and the angle of view. The calculated kernelgenerated on an as needed basis may be acquired or a plurality of thecalculated kernels that are pre-generated and stored may also beacquired. When the calculated kernel is generated, the image processingoperation section 109 functions as a compound kernel acquisition sectionin the present invention. Hereinafter, the blur kernel, the restorationkernel, or the calculated kernel is simply referred to as the kernel.

Next, in Step S302, the calculation of a kernel similarity is performedfor each of the areas to be processed. The kernel similarity is aparameter indicative of the similarity of the kernel assigned to thearea to be processed. When a new area is generated in Step S303, thedegradation amount of the image quality is determined by using thesimilarity.

A first example of the kernel similarity will be given. The firstexample is an example in which the similarity is calculated based on thesize and shape of the kernel. The similarity of the size of the kernelcan be determined by the area of the kernel, the distance from thecenter point thereof, and the radius thereof. For example, the virtualcenter of gravity of each of the kernels is calculated, the averagedistance from the center of gravity as the center point to the edge iscalculated, and the similarity of the kernels may be determined by usingthe average distance. In addition, matching of the shapes of the kernelsis performed and a parameter indicative of the similarity of the shapesmay be calculated.

Next, a second example of the kernel similarity will be given. Thesecond example is an example in which the frequency characteristic ofthe kernel is used. For example, the kernel is subjected to Fouriertransformation and the similarity is calculated by using an modulationtransfer function (MTF). The MTF represents fidelity in reproduction ofthe contrast of the subject on an image plane as the spatial frequencycharacteristic (hereinafter the MTF means the MTF in the image pickupsystem). In the second example, a kernel similarity I is calculated byFormula 1:

$\begin{matrix}\left\lbrack {{Math}\mspace{14mu} 1} \right\rbrack & \; \\{{I = {\int\limits_{\mu_{1}}^{\mu_{2}}{M\; T\;{F(\mu)}{{\mathbb{d}\mu}.}}}}\;\;} & {(1)\mspace{11mu}}\end{matrix}$

Herein, μ represents the spatial frequency, μ₁ and μ₂ are constants forsetting the upper and lower limits of the spatial frequency, and I isobtained by integrating the MTF with the spatial frequency. In thecalculation, the image data may be subjected to Fourier transformationand the frequency characteristic of the image data may also be used.

The kernel similarity I described above can be used as the referencewhen the similarities of a plurality of the kernels associated withvarious distances and angles of view are compared with each other. Notethat the first and second examples may be used in combination. Forexample, the calculated similarity may be weighted and added.

In Step S303, a new area is determined by using the kernel similarity Icalculated in Step S302. The new area determined in this step is an areaobtained by merging the areas to be processed having similar kernelsimilarities I.

More specifically, a threshold T of the image quality degradation amountis prepared, and a plurality of new areas each in which the changeamount of I is not more than T are determined. The area may bedetermined in any manner as long as the change amount of the kernelsimilarity is not more than T. The newly determined area serve as anarea in which the image processing is performed by using the identicalkernel. By limiting the change of the kernel similarity in the area to aspecific value, it is possible to limit the degradation of the imagequality caused by the blurring processing to a specific amount. Notethat the threshold T of the image quality degradation amount may beincluded in the parameters inputted in Step S202, and a value storedtherein may also be used.

A description will be given by using a simple specific example. FIG. 6show the result of the processing in Step S203. FIG. 6A shows the mapbefore the processing in Step S203 is performed. Different kernels areassigned to the individual square areas shown in FIG. 6A.

FIG. 6B shows the map after the processing in Step S203 is performed.For example, in a case where it is determined that, with regard to thekernels included in an area 601, the change amount of the kernelsimilarity I is not more than the threshold, an area 602 as a new areais generated. The area 602 serves as the area where the image processingis performed by using the identical kernel. By performing the processingin Step S203, it is possible to reduce the number of kernels used in theimage processing while limiting the degradation of the image quality tothe specific amount.

In Step S204, the blur kernel corresponding to the new area generated inStep S203 is generated. Specifically, similarly to Step S301, the blurkernel associated with the image pickup conditions, the object distanceand the angle of view according to the optical configuration, and theparameter is generated for each area. In a case where there is a blurkernel that is pre-calculated and stored, the blur kernel may be used.

In addition, with regard to the PSF for generating the blur kernel, thePSF that differs depending on the parameter may be pre-stored. Thecorrespondence to the area, i.e., only elements that determines the PSFsuch as the distance and the angle of view are stored and the blurkernel may be generated by generating the PSF on an as-needed basis. Bythe processing of the present step, the blur kernel corresponding to thenewly generated area is determined.

In Step S205, as the blur kernel generated in Step S204, the restorationkernel generation section 108 generates the restoration kernelcorresponding to the new area generated in Step S203. Specifically,similarly to Step S301, the restoration kernel generation section 108generates the restoration kernel determined by the image pickupconditions and the optical configuration, or its replacement for eacharea. In a case where a restoration kernel is pre-calculated and stored,the restoration kernel may be used.

In the present embodiment, although the restoration kernel is generatedby using the pre-stored PSF, the correspondence to the area, i.e., onlyelements that determines the PSF such as the distance and the angle ofview are stored and the restoration kernel may be generated bygenerating the PSF on an as-needed basis. By the processing of thepresent step, the restoration kernel corresponding to the newlygenerated area is determined.

In Step S206, the image processing operation section 109 performs theimage processing by using the image data stored in the image memory 104,the image area newly generated in Step S203, the blur kernel generatedin Step S204, and the restoration kernel generated in Step S205. Theimage processing performed in the present step is processing thatperforms the image restoration and the blurring of the target image.

Specifically, a convolution calculation of the image data and the blurkernel is performed, and a deconvolution calculation of the restorationkernel is performed. Although there are various methods of thedeconvolution calculation, the simplest example of the method will bedescribed herein. First, the deconvolution calculation is defined asFormula 2. Note that P represents the image data in a given image area,B represents the corresponding blur kernel, and S represents thecorresponding restoration kernel:

[Math 2](P

B)(x,y)=∫∫P(x′,y′)B(x−x′,y−y′)dx′dy′.  (2)

This is the calculation for one pixel in the image area. That is, theblurring processing of the image can be performed by repeatedlyperforming the calculation on all pixels in the image area. Herein,since the processing can be performed by using the identical kernel inthe area newly generated in Step S204, the calculation can be performedby using a convolution theorem as in Formula 3:

[Math 3](P

B)(x,y)=IFT{FT[P(x,y)]·FT[B(x,y)]}.  (3)

Herein, FT represents two-dimensional Fourier transformation, and IFTrepresents two-dimensional inverse Fourier transformation. In a casewhere the image having been subjected to the image restoration and theblurring is generated, the calculation may be performed according toFormula 4:

$\begin{matrix}{\mspace{79mu}\left\lbrack {{Math}\mspace{14mu} 4} \right\rbrack} & \; \\{{\left( {P \otimes B \otimes \frac{1}{S}} \right)\left( {x,y} \right)} = {I\; F\; T{\left\{ {F\;{{T\left\lbrack {P\left( {x,y} \right)} \right\rbrack} \cdot F}\;{{T\left\lbrack {B\left( {x,y} \right)} \right\rbrack}/F}\;{T\left\lbrack {S\left( {x,y} \right)} \right\rbrack}} \right\}.}}} & (4)\end{matrix}$

By performing the above calculation on all of the image areas generatedin the area generation section 106, it is possible to give the effectsof the image restoration and the blurring to the inputted image data.Note that, in order to use other methods of the deconvolutioncalculation, the image data to which the blurring effect by theconvolution calculation of the image data and the blur kernel is alreadygiven may be used. In addition, as described above, the convolutioncalculation of the image data may be performed by using the calculatedkernel.

Subsequently, the image output is performed in Step S207, and the imageto which the effects of the image restoration and the blurring are givenis stored in the processed image memory section 110.

Thus, according to the image processing apparatus according to thepresent embodiment, even when the inputted image is not the image thatis perfectly focused, it is possible to obtain the blurred image havinghigh image quality. In addition, it is possible to calculate theshift-invariant area in an arbitrary image. With this, it is possible toreduce the number kernels to be used for the image restoration whilemaintaining necessary and sufficient image quality, and apply theconvolution theorem. As a result, it is possible to provide the blurringeffect having high image quality and significantly reduce thecalculation amount.

(Second Embodiment)

An image pickup apparatus according to a second embodiment is an imagepickup apparatus including the image processing apparatus 1 according tothe first embodiment. FIG. 7 shows the configuration of the image pickupapparatus according to the present embodiment. An image pickup apparatus4 is typically a digital still camera, a digital video camera, or thelike.

In FIG. 7, the reference numeral 400 denotes an image pickup lens thatguides a subject light to an image pickup element 402. The referencenumeral 401 denotes an exposure control element including a diaphragmand a shutter. The subject light having entered via the image pickuplens 400 enters into the image pickup element 402 via the exposurecontrol element 401. The image pickup element 402 is an image pickupelement that converts the object light to an electric signal and outputsthe electric signal, and is constituted by an image sensor such as a CCDor a CMOS.

An image-forming circuit 403 is a circuit for digitizing and visualizingan analog output signal from the image pickup element 402, and outputs adigital image. The image-forming circuit 403 is constituted by ananalog/digital conversion circuit, an auto gain control circuit, an autowhite balance circuit, a pixel interpolation processing circuit, and acolor conversion processing circuit that are not shown, and the imageprocessing apparatus 1 according to the first embodiment.

An exposure control section 404 is means for controlling the exposurecontrol element 401. A range finding control section 405 is means forcontrolling focusing of the image pickup lens 400. The exposure controlsection 404 and the range finding control section 405 are controlled byusing, e.g., a through-the-lens method (TTL) method (a method in whichexposure and focusing are controlled by measuring light actually passingthrough an image pickup lens).

A system control circuit 406 is a circuit that controls the operation ofthe entire image pickup apparatus 4. A memory 407 is a memory that usesa flash ROM or the like that records data for the operation control andprocessing programs in the system control circuit 406. A non-volatilememory 408 is a non-volatile memory such as an EEPROM for storinginformation items such as various adjustment values and the like thatcan be electrically erased or recorded.

A frame memory 409 is a memory that stores images generated in theimage-forming circuit 403 that are equivalent to several frames. Amemory control circuit 410 is a circuit that controls an image signalinputted to or outputted from the frame memory 409. An image outputsection 411 is means for displaying the image generated in theimage-forming circuit 403 on an image output apparatus that is notshown.

In the second embodiment, the image obtained by the image pickup element402 is inputted to the image-forming circuit 403 including the imageprocessing apparatus according to the first embodiment. That is, theimage data acquired by the image pickup element 402, i.e., datarepresenting the two-dimensional brightness value is the input to theimage data input section 103 in the first embodiment. It is possible toacquire the distance information inputted to the image data inputsection 103 by processing by the known DFD method executed by theimage-forming circuit 403. It is possible to acquire the parametersrelated to the image pickup conditions inputted to the parameter inputsection 101 from the system control circuit 406.

The contents of the image restoration and the blurring processingperformed by the image-forming circuit 403 are the same as thoseperformed by the image processing apparatus 1 according to the firstembodiment.

According to the second embodiment, it is possible to provide the imagepickup apparatus capable of performing the blurring having high qualityon the picked-up image by using the image processing apparatus accordingto the first embodiment.

(Third Embodiment)

A third embodiment is an embodiment in which the kernel similarity I inthe first embodiment is determined based on a visual spatial frequencycharacteristic in addition to the frequency characteristic of thekernel.

The present embodiment is different from the first embodiment only inthe calculation method of the kernel similarity in Steps S301 and S302.The configuration and the processing method are otherwise the same asthose in the first embodiment.

In the present embodiment, the kernel similarity is calculated by usingthe visual spatial frequency characteristic in addition to the MTF ofeach of the blur kernel and the restoration kernel. As an example, avalue called a cascaded modulation transfer acutance (CMTA) is used. TheCMTA is described in detail in “TAKAGI Mikio, SHIMODA Haruhisa,“Handbook of Image Analysis (Revised Edition)”, University of TokyoPress, p. 77-78, 2004”.

It is possible to determine the shift-invariant area by using the methoddescribed in the first embodiment. However, it is known that human sighthas a distinct spatial frequency characteristic, and a spatial frequencyhaving a large reaction to the change of the spatial frequency and aspatial frequency having a small reaction to the change of the spatialfrequency are present. Accordingly, by weighting the MTF of the kernelby using the visual spatial frequency characteristic, it becomespossible to determine the kernel similarity more accurately in the areawhere a human being cannot perceive the change.

The MTF for the visual spatial frequency is known as a contrastsensitivity function (CSF). The CSF is obtained by modeling of a visualcontrast sensitivity in consideration of characteristics of a low-passfilter in the image-forming system of an eyeball and characteristics ofa band-pass filter in the signal processing system from a retina to abrain. An example of the CSF is represented by Formula 5:

[Math 5]CSF(f)=a·f·exp(−b·f).  (5)

Herein, f is the spatial frequency, and is represented by a unit(cycle/deg) representing the number of times of viewing a contraststripe per degree of human visual angle. Although a is often set to 75and b is often set to 0.2, the values are not fixed and are changeddepending on various conditions related to an evaluation environment.

A factor obtained by normalizing the integration value of the product ofthe CSF and the MTF of each of the blur kernel and the restorationkernel in a given spatial frequency range is called a subjective qualityfactor (SQF). Herein, the MTF of the blur kernel in the presentembodiment is assumed to be an MTFb, while the MTF of the restorationkernel in the present embodiment is assumed to bean MTFs. The SQF isobtained by (Formula 6) dividing the integration value (integratedvalue) of the product of the spatial frequency characteristic of thekernel and the visual spatial frequency by the integration value(integrated value) of the visual spatial frequency. Note that frepresents the spatial frequency and f₁ and f₂ represent constants forsetting the upper and lower limits of the spatial frequency:

$\begin{matrix}\left\lbrack {{Math}\mspace{14mu} 6} \right\rbrack & \; \\{{S\; Q\; F} = {\frac{\int\limits_{f_{1}}^{f_{2}}{M\; T\;{{F_{b}(f)}/M}\; T\;{{F_{s}(f)} \cdot C}\; S\;{F(f)}{\mathbb{d}f}}}{\int\limits_{f_{1}}^{f_{2}}{C\; S\;{F(f)}{\mathbb{d}f}}}.}} & (6)\end{matrix}$

The CMTA is obtained by making the SQF linear to the sense of a humanbeing by using Weber-Fechner's law and normalizing the SQF to 100.Weber-Fechner's law mentioned herein is a law that the sense of a humanbeing is proportional to a logarithm of a stimulation, and the CMTA canbe formulated as Formula 7. By using the CMTA as the kernel similarityI, it is possible to determine the shift-invariant area in which thevisual spatial frequency characteristic is reflected:

[Math 7]CMTA=100+66 log₁₀(SQF).  (7)

Thus, according to the image processing apparatus according to thepresent embodiment, by using the evaluation value using the CMTA, it ispossible to identify the range of the image quality change that a humanbeing cannot perceive and determine the shift-invariant area moreaccurately. That is, it is possible to reduce the number of kernels forthe image restoration while maintaining the necessary and sufficientimage quality for the human sight, and apply the convolution theorem. Asa result, as compared with the first embodiment, it is possible toobtain the blurring effect having high image quality while achieving areduction in calculation amount.

The description of the embodiments is illustrative for the descriptionof the present invention, and the present invention can be implementedby appropriately changing or combining the embodiments without departingfrom the gist of the invention. The present invention can be implementedas an image processing method including at least a part of the aboveprocessing, or implemented as an image processing program that causesthe image processing apparatus to execute the method. The aboveprocessing and means can be freely combined unless technicalcontradiction arises.

Aspects of the present invention can also be realized by a computer of asystem or apparatus (or devices such as a CPU or MPU) that reads out andexecutes a program recorded on a memory device to perform the functionsof the above-described embodiment(s), and by a method, the steps ofwhich are performed by a computer of a system or apparatus by, forexample, reading out and executing a program recorded on a memory deviceto perform the functions of the above-described embodiment (s). For thispurpose, the program is provided to the computer for example via anetwork or from a recording medium of various types serving as thememory device (e.g., non-transitory computer-readable medium).

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2012-114737, filed on May 18, 2012, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus that blurs apicked-up image based on distance information on a subject, comprising:a restoration kernel acquisition section that acquires a restorationkernel as a kernel for eliminating degradation of an image, wherein therestoration kernel acquisition section acquires the restoration kernelassociated with a distance and an angle of view corresponding to an areato be processed on the picked-up image; a blur kernel acquisitionsection that acquires a blur kernel as a kernel for blurring the image;a compound kernel acquisition section that acquires a compound kernelobtained by merging the restoration kernel with the blur kernel; and animage processing section that eliminates the degradation of thepicked-up image and blurs the picked-up image by using the compoundkernel.
 2. The image processing apparatus according to claim 1, whereinthe blur kernel acquisition section acquires the blur kernel associatedwith either of the distance or the angle of view corresponding to thearea to be processed on the picked-up image.
 3. The image processingapparatus according to claim 1, wherein the image processing sectionassigns the compound kernel to the area to be processed on the picked-upimage, and performs blurring processing including image restoration byusing the assigned compound kernel.
 4. The image processing apparatusaccording to claim 3, further comprising: an image area extensionsection that determines an extended area as an area including at leastone area to be processed, wherein the image area extension sectiondetermines a similarity of the compound kernel assigned to each area tobe processed to merge, into one extended area, the areas to be processedcorresponding to a plurality of the compound kernels that are determinedto be similar to each other, and the image processing section performsthe blurring processing including the image restoration on the extendedarea by using an identical compound kernel.
 5. The image processingapparatus according to claim 4, wherein the image area extension sectiondetermines the similarity based on a size or a shape of the compoundkernel assigned to the area to be processed.
 6. The image processingapparatus according to claim 4, wherein the image area extension sectiondetermines the similarity based on a spatial frequency characteristic ofthe compound kernel assigned to the area to be processed.
 7. The imageprocessing apparatus according to claim 6, wherein the image areaextension section determines the similarity further based on a visualspatial frequency characteristic.
 8. The image processing apparatusaccording to claim 7, wherein the image area extension sectiondetermines the similarity based on a ratio between an integrated valueof a product of the visual spatial frequency characteristic and thespatial frequency characteristic of the compound kernel assigned to thearea to be processed, and an integrated value of the visual spatialfrequency.
 9. An image processing apparatus that blurs a picked-up imagebased on distance information on a subject, comprising: a restorationkernel acquisition section that acquires a restoration kernel as akernel for eliminating degradation of an image; a blur kernelacquisition section that acquires a blur kernel as a kernel for blurringthe image; and an image processing section that eliminates thedegradation of the picked-up image and blurs the picked-up image byusing the restoration kernel and the blur kernel, wherein therestoration kernel acquisition section acquires the restoration kernelassociated with a distance and an angle of view corresponding to an areato be processed on the picked-up image, and the blur kernel acquisitionsection acquires the blur kernel associated with the distance and theangle of view corresponding to the area to be processed on the picked-upimage.
 10. An image pickup apparatus comprising: image pickup unit; andthe image processing apparatus according to claim
 1. 11. Anon-transitory recording medium recording thereon an image processingprogram for causing an image processing apparatus that blurs a picked-upimage based on distance information on a subject to execute the stepsof: acquiring a restoration kernel as a kernel for eliminatingdegradation of an image, wherein the acquired the restoration kernel isassociated with a distance and an angle of view corresponding to an areato be processed on the picked-up image; acquiring a blur kernel as akernel for blurring the image; acquiring a compound kernel obtained bymerging the restoration kernel with the blur kernel; and eliminating thedegradation of the picked-up image and blurring the picked-up image byusing the compound kernel.
 12. A non-transitory recording mediumrecording thereon an image processing program for causing an imageprocessing apparatus that blurs a picked-up image based on distanceinformation on a subject to execute the steps of: acquiring arestoration kernel as a kernel for eliminating degradation of an image;acquiring a blur kernel as a kernel for blurring the image; andeliminating the degradation of the picked-up image and blurring thepicked-up image by using the restoration kernel and the blur kernel,wherein the restoration kernel associated with a distance and an angleof view corresponding to an area to be processed on the picked-up imageis acquired in the step of acquiring a restoration kernel, and the blurkernel associated with the distance and the angle of view correspondingto the area to be processed on the picked-up image is acquired in thestep of acquiring a blur kernel.